Skip to content

feat(data): add reproducible adaptation dataset contracts - #151

Closed
alpha912 wants to merge 2 commits into
NandhaKishorM:mainfrom
MycelAI:alpha912/fix/decision-pipeline-data
Closed

alpha912 wants to merge 2 commits into
NandhaKishorM:mainfrom
MycelAI:alpha912/fix/decision-pipeline-data

Conversation

@alpha912

@alpha912 alpha912 commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Prepare labelled domain-adaptation data with reproducible boundaries between training, temperature calibration, policy selection and final evaluation. Random row splits can put copies, translations or revisions of the same example on both sides of an evaluation; this opt-in data contract keeps declared and detected families together and preserves the provenance needed to audit the split.

Changes

  • Validate existing choice, score and noul question definitions and strict hard targets, including partially labelled examples and source provenance.
  • Merge declared families, normalized exact duplicates and repeated source identities transitively before deterministic group assignment. Reject contradictory labels for duplicate states.
  • Preserve official source test sets through recorded reservations that propagate to whole connected families. Reject conflicting reservations; zero-fraction roles remain reserved.
  • Persist and verify UTF-8 datasets and manifests, including rendering-sensitive question/state order. Report row counts separately from connected-group counts.
  • Select one representative per group and language using IDs and the fixed seed, independently of target values or model scores.

Testing

Additional validation on the minimum supported Python version: all 21 data-contract tests pass on Windows with Python 3.10.11, CPU Torch 2.14.0, Transformers 5.17.0 and NumPy 2.2.6. The isolated test run used synthetic/local fixtures with Hub access disabled; it establishes compatibility, not model quality.

All checks ran locally without downloading pretrained weights or running training.

  • Windows and Linux, Python 3.12.11: all 21 contract tests pass, including fixed-family reservations and a hash-rounding boundary that must not select a zero-fraction role.
  • Existing criteria (34), decision-model (3), and packaging (9) checks pass on both hosts; routing (150) also passed on Windows.
  • Ruff, byte compilation and git diff whitespace checks pass.
  • Coverage includes typed-label errors, partial labels, ordered hashes, transitive grouping, cross-revision identities, conflicting duplicates, independent representatives, manifest mismatches, Unicode round trips and atomic publication failures.

Notes for reviewer

This is the data foundation for a larger domain-adaptation/selective-decision workflow. It preserves the SDK, adds no dependency, and leaves inference, training, calibration fitting and GitHub workflows unchanged. It complements #19 and #53. Existing manifests without reservations remain verifiable.

Adapters must identify translations, conversations and semantic near-duplicates that exact matching cannot infer. Language and license metadata require source audits. Hash verification establishes consistency, not authenticity. Fractions are expected proportions; small or skewed splits can lack evidence.

These tests establish data/configuration behavior only. No model-accuracy, calibration-quality or automation-coverage improvement is claimed.

Independent calibration and evaluation need stable provenance and family boundaries. Preserve rendered schema order, reject conflicting duplicate labels, and expose one representative per group so repeated tickets cannot inflate statistical evidence.
Official test families must stay held out when derived variants or duplicate links are present. Record reservations in the verified manifest and keep zero-probability roles reserved.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, keeping families of near-duplicate examples on one side of a split is exactly the discipline that stops calibration looking better than it is. #186 showed that problem in our own notebook today. This is part of a larger series (#151, #152, #159, #164, #165, #167), and it's a lot to review at once. Could you tell me which one piece you'd most like to land first? A single focused PR with a short example is what I can review properly.

@alpha912
alpha912 marked this pull request as draft September 23, 2026 17:29
@alpha912

Copy link
Copy Markdown
Contributor Author

Thanks for the clear steer. Of the six, please prioritize #151 first, once I mark it ready again. I have put #151 and #152 back in draft; #159, #164, #165 and #167 will stay draft.

The smallest useful slice is the family-preserving split. For example, an English lost-card request and its Arabic translation from the same source family must stay on one side of the train/calibration/policy/test boundary. That is the data condition the later risk claims depend on.

I'll tighten #151 around that slice, put the example and local verification up front, and request review only when it is small enough to review properly. No action needed on the rest of the stack now.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for the clear plan. I've thought about scope: the open-source package is staying focused on fast local inference, so adaptation, training and deployment tooling like this series is a better fit as a standalone project than inside laya/. I'd rather say that now than after you've tightened it further. Thank you for the careful work on family-preserving splits.

@alpha912

Copy link
Copy Markdown
Contributor Author

Thanks for the clear scope decision. I will close the six adaptation draft PRs (#151, #152, #159, #164, #165, #167) to clear the review queue and preserve the work locally. I will finish the frozen evaluation before considering a standalone project. No further review is needed on this series.

@alpha912 alpha912 closed this Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants